The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence

We replicate and reanalyse the most influential study of microcredit impacts (M. M. Pitt & S. R. Khandker’s, ‘The impact of group-based credit on poor households in Bangladesh: Does the gender of participants matter?’, published in the Journal of Political Economy, 106, 1998). That study was celebrated for showing that microcredit reduces poverty, a much hoped for possibility (though one not confirmed by recent randomised controlled trials). We show that the original results on poverty reduction disappear after dropping outliers, or when using a robust linear estimator. Using a new program for estimation of mixed process maximum likelihood models, we show how assumptions critical for the original analysis, such as error normality, are contradicted by the data. We conclude that questions about impact cannot be answered in these data. Over the last few decades, microcredit has captured millions of customers, billions of dollars in financing, a Nobel Prize, and the imagination of the global public. Many have seen microcredit as lifting families out of poverty, especially when lent to women. The movement owes its strength in part to an early literature based on observational data that shows strong positive impacts. The most prominent studies in this literature took place in the leading nation of microcredit, Bangladesh. More recently, muted results from randomised trials in India, the Philippines, and elsewhere are prompting second thoughts. The sharp contradiction between the old and new studies raises questions. Has the impact of microcredit varied over time and place? Is the key that the Bangladesh studies were longer term? Or is the difference in methods? Some of those questions cannot be answered without replicating studies and extending them to gauge robustness. Toward that goal, we revisit the most-cited evaluation of the impacts of microcredit, Pitt and Khandker (PK) (1998), which is based on a structural model that disaggregates impacts by gender and relies in part on assumptions akin to regression discontinuity design. The study is notable for its historical place in the literature, its long time frame, and its relevance to the continuing public controversy over the efficacy of microcredit. Grameen Bank founder Muhammad Yunus once regularly claimed, in an extrapolation from coefficient estimates in PK, that ‘In a typical year 5 per cent of Grameen borrowers ... rise above the poverty level.’ PK remains the single most cited empirical study of microcredit, with 890 cites on Google Scholar as of 17 June 2013. PK attacks selection bias through an innovative and complex limited-information maximum likelihood (LIML) framework. While questions have been raised about the robustness of results to alternative estimation methods (Chemin, 2008; Duvendack & Palmer-Jones, 2012; Morduch, 1998), Pitt (1999, 2012) has strongly defended PK against such criticisms. Our close replication of the original methods helps resolve several outstanding disputes. Correspondence Address: Jonathan Morduch, Financial Access Initiative, NYU Wagner Graduate School, New York University, 295 Lafayette Street, New York, 10012, USA. Email: [email protected] The Journal of Development Studies, 2014 Vol. 50, No. 4, 583–604, http://dx.doi.org/10.1080/00220388.2013.858122 © 2013 Taylor & Francis D ow nl oa de d by [ 73 .1 97 .1 72 .2 14 ] at 1 1: 14 2 8 Fe br ua ry 2 01 5 We find several problems in PK. The PK finding that microcredit reduced poverty especially when given to women is robust to fixes for some but not all of these problems. A seemingly innocuous choice in imputation for the log of 0 in the borrowing variables leaves the effect sizes unidentified. A discontinuity in credit availability, asserted as the basis for quasi-experimental identification, is missing in the data. By the same token, in the treatment group, but not the control group, many borrowers above the official wealth limit for eligibility are coded as eligible, suggesting endogeneity in this ‘intention to treat’ variable. Finally, the estimator is bimodal on the PK data, producing a mode with a positive impact estimate and a mode with a negative estimate. One cause appears to a long right tail in household consumption, the dependent variable of primary interest, which itself violates a normality assumption. Dropping the 16 rightmost observations in this tail, 0.4 per cent of the sample, causes the two modes to collapse into one near zero – that is, to erase the PK finding. Instrument weakness may also play a role, as the bimodality appears to arise from the subsample in which the instruments are least able to differentiate impacts by gender. This article is part of a debate that is notable for its length, complexity, and intensity (Morduch, 1998; Pitt 1999, 2011; Roodman &Morduch, 2011; PK, 2012). In our view, this odyssey offers two lessons for the social sciences in general. The first is about the limitations of the traditional journal review process and the value of replication in going beyond it. PK (1998) was published in the prestigious Journal of Political Economy after a rigorous review process. Still, journal editors and referees are limited in their abilities to fully assess studies. The anonymity that protects referees also limits their ability to communicate with authors to gain clarification. Referees’ limited time and attentionmeans that they rarely look at data and computer code to probe statistics on their own. They may not have visited the places under study, or have read more than a small slice of the cited literature. Referees focus on coherence, completeness, relevance, and originality. Their work goes far, but it is not a substitute for re-analysis. Thework of clarification, replication, refutation, and extension is necessarily left to others, but scholars seldom directly replicate the work of others, especially in development studies, where the abundance of opportunities to break new ground imposes high opportunity costs on replication. The second lesson is about the value of open data and code sharing. Morduch began his dialogue with PK in 1998. The present phase beganwith exchanges in 2007.While underlying survey data was shared early on, only in 2011 did a file become publicly available that included all constructed variables needed to run the regression of primary interest (Pitt, 2011). Its release was provoked by the first edition of this analysis, which itself entailed significant effort. Meanwhile, the original computer code is reportedly lost. Transparency in data and code could have shaved a decade off the scrutinising of these influential, policy-relevant results. Such transparency is still far from the norm in the social sciences. This article runs as follows. Section 1 describes the PK estimator and explores its assumptions. Section 2 replicates the ‘headline’ regression relating to household consumption. Section 3 demonstrates four concerns about the estimator and tests fixes where possible. Section 4 analyses regressions of non-consumption outcomes. Section 5 concludes. 1. The Econometrics of PK 1.1. The Estimation Problem PK analyse data from surveys of 1,798 households in 87 randomly selected villages within a randomly selected 29 of Bangladesh’s 391 upazillas. Surveyors visited the households in 1991–1992 after each of the three main rice seasons: aman (December–January), boro (April–May), and aus (July–August). Only 29 households attrited by the third round. Ten of the 87 villages had male microcredit borrowing groups, 22 had female groups, and 40 had both. All groups were single-sex. Credit programmes of three institutions were evaluated: the Grameen Bank; a large non-governmental group called BRAC; and the official Bangladesh Rural Development Board (BRDB). According to PK (1998, p. 959), all three programmes essentially set eligibility in terms of land ownership: only functionally landless households, defined as those owning half an acre or less, could borrow. For statistical precision, the surveyors oversampled households poor enough to be targeted for microcredit. Since sampling on the basis of eligibility can bias results, PK incorporate sampling weights constructed from village censuses. 584 D. Roodman & J. Morduch D ow nl oa de d by [ 73 .1 97 .1 72 .2 14 ] at 1 1: 14 2 8 Fe br ua ry 2 01 5 PK study six outcomes. Two are household-level: per capita consumption and female-owned nonland assets. Four are individual level: male and female labour supply and school enrolment of girls and boys. For each outcome, the three-way split by credit supplier and the two-way split by sex lead to six parameters of interest, the impact coefficients on credit by lender and gender. A central feature of the estimation problem is that the credit variables are at once presumed endogenous and bounded from below. Meanwhile, all of the outcomes except log household consumption are themselves bounded or binary. PK therefore estimate the impact parameters using a LIML framework that models the limited nature of all the endogenous variables. Each fitted model contains equations for the outcome variable of interest as well as for female borrowing and male borrowing. The outcome is variously modelled as Tobit, probit, or linear and unbounded. 1.2. The Estimator To state the PK model, we first need to formally describe access to credit. Let pf and pm be dummies indicating whether credit groups composed of females or males are operating in the village of a given household or household member; they capture credit availability by gender. Let e be a dummy for whether a household or household member is deemed eligible for a microcredit programme, regardless of whether any borrowing groups operate in its village. Then the credit choice variables, indicating whether members of each sex can borrow, are cf 1⁄4 pf e cm 1⁄4 pme A central contention in PK is that cf and cm are exogenous and excludable. This allows the availability of microcredit to be thought of as ‘intent to treat’, and to instrument for actual uptake, or ‘treatment’. The contention that cf and cm are good instruments is based in part on the idea that e depends on the discontinuous half-acre eligibility rule. Since we focus on the outcome log per capita household consumption, the basis of PK’s influential finding that microcredit reduces poverty, we take the outcome variable yo to be continuous and unbounded. Let yf (ym) be the logarithm of total microcredit borrowings of all females (males) in a household. 6 Let yfm ; yf 1; yf 2; yf 3; ym1; ym2; ym3 0 be the six credit variables, disaggregated by lender and gender. And let x be a vector of controls that includes the eligibility dummy e, log landholdings, household characteristics, village and survey round dummies, and a constant. Let Ct be the credit censoring threshold, the minimum observable log borrowing amount among borrowers. If there is no borrowing, the household gets Cv, the censoring value for log borrowing assigned by the researcher (necessary since log 0 is undefined). Then the PK estimation model, fit with maximum likelihood (ML), can be written as: yo 1⁄4 y0fmδþ x 0 βo þ o y f 1⁄4 x 0 βf þ f if cf 1⁄4 1 y m 1⁄4 x 0 βm þ m if cm 1⁄4 1 yf 1⁄4 y f if cf 1⁄4 1 and y f Ct Cv otherwise ym 1⁄4 y m if cm 1⁄4 1 and y m Ct Cv otherwise ; o; f ; m 0 |x, iidN 0;S ð Þ (1) where S is a 3 × 3 positive-definite symmetric matrix. The Impact of Microcredit on the Poor in Bangladesh 585 D ow nl oa de d by [ 73 .1 97 .1 72 .2 14 ] at 1 1: 14 2 8 Fe br ua ry 2 01 5 The PK model is unusual in several respects. The three main equations include the same exogenous regressors, x: seemingly, no instruments are excluded. The exogeneity of cf and cm is the asserted basis for identification, yet those dummies do not seem to serve as instruments. The credit equations’ samples are restricted, so the number of equations in the model varies by observation. The outcome equation contains six endogenous credit variables, yfm, but the model includes just two instrumenting equations. The instrumenting stage is modelled as censored, which forces the unusual distinction between the censoring threshold, relevant for the Tobit modelling in the credit equations, and the censoring value, relevant for the treatment of credit on the right side of the outcome equation. And while PK set out to exploit a discontinuity in access to credit, the sample is not concentrated as in conventional Regression Discontinuity Design around the half-acre mark, but spans from a de minimus 0.1 acres to 5 acres. This wide bandwidth necessitates a parametric approach. 1.3. A Closer Look at Assumptions A key to understanding some of these unusual characteristics is to note that the last line of (1) elides a complexity. The yf and ym equations are not defined over the full sample, so f , m, and the joint distribution N 0;S ð Þ are not either. So to state the distributional assumption precisely, we distinguish the four possible cases of credit availability by gender. We use combinations of o, f , and m subscripts to denote subvectors of and submatrices of S corresponding to combinations of the equations for the outcome, female credit, and male credit. A precise statement of the distributional assumption (not spelled out in PK) is then: ojx , N 0;So ð Þwhen cf 1⁄4 0; cm 1⁄4 0 of x , N 0;Sof when cf 1⁄4 1; cm 1⁄4 0 omjx , N 0;Som ð Þwhen cf 1⁄4 0; cm 1⁄4 1 ofm x , N 0;Sofm when cf 1⁄4 1; cm 1⁄4 1 where ofm ; andSofm ; S : Every case implies "ojx, N 0;So ð Þ: Thus ojx; cf ; cm , N 0;So ð Þ (2) That is, knowing credit availability by gender tells us nothing about the distribution of o. This is how the identification strategy implies and requires that credit choice is exogenous. One can gain further intuition by innocuously inserting cf and cm into the latent credit equations in (1): y f 1⁄4 cf x 0 βf þ f if cf 1⁄4 1 y m 1⁄4 cmx 0 βm þ m if cm 1⁄4 1 (3) This communicates the idea that cf x and cmx are the instruments, being excluded from the yo equation. And since x includes a constant, cf and cm are now seen as instruments too. One important question about the PK estimation model is whether its distributional assumptions must hold strictly for the estimates of δ to be consistent. ML estimation of misspecified models can be consistent for some parameters (White, 1982). For example, linear LIML is naturally derived from a model that assumes iid normal errors, but is consistent under substantial violations of that assumption: errors need not be normal, and they need only be uncorrelated with the instruments, not independent (Anderson & Rubin, 1950). The nonlinearities in the PK estimator turn out to make it less robust to such violations. For example, the estimator is inconsistent if o has skewness, as simulations in the appendix demonstrate. Similarly, if the first-stage Tobit models are not exactly correct, then the estimator should be presumed inconsistent (Angrist & Krueger, 2001). In contrast, a linear instrumental variables (IV) estimator defined along the lines of (3) – instrumenting with cf x and cmx and dispensing with the Tobit 586 D. Roodman & J. Morduch D ow nl oa de d by [ 73 .1 97 .1 72 .2 14 ] at 1 1: 14 2 8 Fe br ua ry 2 01 5 modelling of borrowing – is consistent regardless of the true functional form and error distribution of the first stage (Kelejian, 1971). The PK specifications that include village dummies in x, among them the headline regression suggesting that microcredit reduces poverty, are akin to the difference-in-differences (DID) estimator with controls. The two dimensions of difference are the eligibility of a household for microcredit (indicated by e) and the availability of microcredit in a village (pf and pm). As in DID, identification comes from variation associated with the excluded products pf e and pme conditional on the included factors pf , pm, and e (pf and pm being controlled for through the village dummies). 10 The validity of the exclusion assumption is open to question (Morduch, 1998). For example, in villages where eligible households are relatively well-off, credit group formation may be more likely. In this way, village effects may interact with eligibility to cause outcomes through channels separate from microcredit. 2. Replication Pitt (2011) provides a data set adequate for replicating the PK regression of primary interest, with yo as log per capita household consumption. The first and second moments of regression variables in the Pitt (2011) data closely match those reported in PK – though not exactly (See Tables 1 and 2). The five other PK outcomes are not in the Pitt (2011) data, nor in a set sent earlier to us by Mark Pitt. So we construct those outcomes from the underlying survey data. Among the five, the match is extremely good for male labour supply and boys’ and girls’ school enrolment. It is poorer for female labour supply. But here we have reason to doubt PK’s aggregates. PK (2002, Table 1) reports the same means alongside mathematically incompatible seasonal sub-averages. Finally, the biggest Table 1. Weighted means and standard deviations of individualand household-level right-side variables, first survey round, as reported in PK and in reconstructions Mean Standard deviation PK New PK New Age of all individuals 23 23 18 18 Schooling of individual 5 or above 1.377 1.386 2.773 2.780 Parents of household head own land? 0.256 0.250 0.564 0.559 # of brothers of household head owning land 0.815 0.796 1.308 1.298 # of sisters of household head owning land 0.755 0.737 1.208 1.197 Parents of household head’s spouse own land? 0.529 0.521 0.784 0.780 # of brothers of household head’s spouse owning land 0.919 0.905 1.427 1.421 # of sisters of household head’s spouse owning land 0.753 0.740 1.202 1.195 Household land (in decimals) 76.142 75.883 108.54 107.98 Highest grade completed by household head 2.486 2.479 3.501 3.500 Sex of household head (1 = male) 0.948 0.947 0.223 0.223 Age of household head (years) 40.821 40.803 12.795 12.790 Highest grade completed by any female household member 1.606 1.601 2.853 2.851 Highest grade completed by any male household member 3.082 3.069 3.081 3.77 Adult female not present in household? 0.017 0.017 0.129 0.130 Adult male not present in household? 0.035 0.036 0.185 0.185 Spouse not present in household? 0.126 0.126 0.332 0.332 Amount borrowed by female from BRAC (taka) 350.345 350.369 1,573.65 1,573.63 Amount borrowed by male from BRAC (taka) 171.993 171.973 1,565 1,565 Amount borrowed by female from BRDB (taka) 114.348 114.119 747.301 746.722 Amount borrowed by male from BRDB (taka) 203.25 202.79 1,572.66 1,571.62 Amount borrowed by female from Grameen (taka) 956.159 953.581 4,293.36 4,287.96 Amount borrowed by male from Grameen Bank (taka) 374.383 373.940 2,922.79 2,921.46 Nontarget household 0.295 0.293 0.456 0.455 N = 1,757, First two variables are reconstructed from PK survey data. Remainder are from Pitt (2011). Treats current students as having no years of schooling. The Impact of Microcredit on the Poor in Bangladesh 587 D ow nl oa de d by [ 73 .1 97 .1 72 .2 14 ] at 1 1: 14 2 8 Fe br ua ry 2 01 5